In a conventional distributed system, a distributed job includes a limited number of tasks, and there is a certain dependency between the tasks. Each task is divided into a plurality of task instances. When the job is executed, the data processing is applied to the task instances of each task.
Based on the conventional distributed system, job scheduling mode is usually conducted by a central node (Master) for unified scheduling. For example, assuming that a job has two tasks, namely task 1 and task 2 respectively, the dependency between task 1 and task 2 is that the input of task 2 is the output of task 1. After the job is executed, the central node searches for a task that has no predecessor task. If task 1 is executing firstly, after all the task instances of the task 1 have finished execution, the central node then schedules task 2 to execute.
However, in this conventional job scheduling mode, the amount of data processed and maintained by the central node is massive, which affects the scheduling efficiency and does not make full use of the cluster resources, thereby resulting in the waste of resources.